1. Field of the Invention
The present invention relates generally to image processing for a digital display system, and relates more particularly to an apparatus and method for performing image transforms and multi-frame processing of input images to produce high-quality output images.
2. Discussion of Prior Art
Cathode Ray Tubes (CRTs), used in conventional televisions and computer monitors, are analog devices which scan an electron beam across a phosphor screen to produce an image. Digital image-processing products that enhance display graphics and video on CRTs have been increasingly available, because CRTs can operate with many different input and output data formats. Further, CRTs can display moving images with high quality screen brightness and response. However, CRTs have considerable limitations in such applications as portable flat-screen displays where size and power are important. Additionally, as direct-view CRT display size increases, achieving high image quality across the complete display becomes more difficult and expensive.
Many recent portable and desktop systems include digital displays using liquid crystal displays (LCDs), a term which generally describes flat-panel display technologies and in particular, may include active matrix liquid crystal displays (AMLCDs), silicon reflective LCDs (si-RLCDs), ferroelectric displays (FLCs), field emission displays (FEDs), electroluminescent displays (ELDs), plasma displays (PDs), and digital mirror displays (DMDs).
Compared to traditional CRT displays, LCDs have the advantages of being smaller and lighter, consuming less power, and having discrete display elements which can provide consistent images across the entire display. However, manufacturing LCDs requires special processing steps to achieve acceptable visual quality. Further, large screen direct view LCDs are expensive, and LCDs usually require a display memory.
Both CRT and LCD technologies can provide economical projection-system large screen displays. CRT-based projection systems usually require three CRTs and three projection tubes, one for each of the Red (R), Green (G), and Blue (B) color components. Each tube must produce the full resolution display output at an acceptable brightness level, which makes the tubes expensive. Achieving proper tolerances for mechanical components in projection systems, including alignment hardware and lenses, is also expensive. Consequently, manufacturing CRT-based projection systems is costly. Since CRTs are analog devices, applying digital image-processing techniques to CRT-based systems usually requires a frame buffer memory to effectively represent the digital image data.
Projection display systems also may use transmissive or reflective LCD xe2x80x9cmicrodisplayxe2x80x9d technologies. Achieving the desired full color gamut in LCD-based parallel color projection systems, as in CRT-based projection systems, uses three separate LCD image modulators, one for each of the R, G, and B color components. A single LCD image modulator which produces R, G, and B, either through spatial color filters or with sequential color fields at a sufficiently high rate, can provide a low cost system.
FIG. 1 shows a prior art projection system 150 that includes a light system 100, mirrors 102, 104, 106, and 108, transmissive image modulators 110, 112, and 114, dichroic recombiners 116 and 118, and a projection lens 120. Light system 100 includes an illumination source such as a xenon lamp and a reflector system (not shown) for focusing light.
Mirrors 102, 104, 106, and 108, together with other components (not shown) constitute a separation subsystem that separates the light system 100 output white light beam into color components Red (R), Green (G), and Blue (B). The separation subsystem can also use prisms, including x-cube dichroic prism pairs or polarizing beam splitters.
Each image modulator 110, 112, and 114 receives a corresponding separated R, G, or B color component and functions as an active, full resolution, monochrome light valve that, according to the desired output images, modulates light intensities for the respective R, G, or B color component. Each image modulator 110, 112, and 114 can include a buffer memory and associated digital processing unit (not shown). A projection system may use only one image modulator which is responsible for all three color components, but the three image modulator system 150 provides better chromaticity and is more efficient.
Dichroic recombiners 116 and 118 combine modulated R, G, and B color components to provide color images to projection lens 120, which focuses and projects images onto a screen (not shown).
FIG. 1 system 150 can use transmissive light valve technology which passes light on axis 1002 through an LCD shutter matrix (not shown). Alternatively, system 150 can use reflective light valve technology (referred to as reflective displays) which reflects light off of digital display mirror display (DMD) image modulators 110, 112, and 114. Because each image modulator 10, 112, and 114 functions as an active, full resolution, monochrome light valve that modulates the corresponding color component, system 150 requires significant buffer memory and digital image processing capability.
Because of inherent differences in the physical responses of CRT and LCD materials, LCD-based projection and direct view display systems each have different flicker characteristics and exhibit different motion artifacts than CRT-based display systems. Additionally, an intense short pulse depends on the properties of CRT phosphors to excite a CRT pixel, whereas a constant external light source is intensity modulated during the frame period of an LCD display. Further, LCDs switch in the finite time it takes to change the state of a pixel. Active matrix thin film transistor (TFT) displays, which have an active transistor controlling each display pixel, still require a switching time related to the LCD material composition and thickness, and to the techniques of switching.
Most LCD-based image modulators (such as 110, 112, 114) are addressed in raster scan fashion and each pixel requires refreshing during each display frame interval. Accordingly, every output pixel is written to the display during every refresh cycle regardless of whether the value of the pixel has changed since the last cycle. In contrast, active matrix display technologies and some plasma display panel technologies may allow random access to the display pixels. Other panels use a simpler row-by-row addressing scheme that is similar to the raster scan of a CRT. Additionally, some displays have internal storage to enable output frames to self-refresh based on residual data from the previous output frame.
Field Emission Displays (FEDs) may include thousands of microtips grouped in several tens of mesh cells for each pixel. The field emission cathodes in FEDs can directly address sets of row or column electrodes in FEDs, and FEDs have rapid response times. FEDs can use external mesh addressing for better resolution images, but this requires increased input/output (I/O) bandwidth outside of the FED.
Opto-mechanical systems can provide uniform brightness and high chromaticity for high quality displays. Additionally, high quality projection lens systems can provide bright and uniform images. However, component and assembly tolerances in opto-mechanical systems can result in system imperfections including imprecise image modulator alignment and geometric lens distortion.
Commercially-available digital image processing systems, usually part of an electronic control subsystem, can process analog or digital input data and format the data into higher resolution output modes. These processing systems typically perform operations such as de-interlacing, format conversion and line doubling or quadrupling for interlaced analog input data. Some systems include a decompression engine for decompressing compressed digital data, and input data scaling to match the resolution and aspect ratio to the display device. However, these systems do not perform advanced image processing that is specific to a digital imaging LCD or to the display system. Additionally, these digital image processing systems do not often accommodate digital or compressed digital image data which can include bitstream information for enhanced outputs.
Image sensing algorithms, for example, in remote sensing and computer vision applications, use special sampling and image warping techniques to correct input sensor distortions and to reconstruct images. The technique of super-resolution uses multiple still frame images which include sub-pixel movement, typically from camera movement, to construct a high resolution still frame.
Data compression tools such as those standardized by the Moving Pictures Experts Group (MPEG) can compact video data prior to transmission and reconstruct it upon reception. MPEG-2 can be applied to both standard definition (SDTV) and high definition television (HDTV) in a variety of resolutions and frame rates.
Projecting an image from a projector on a tabletop to a flat screen which is closer to the projector at the screen bottom than the screen top results in an image which is narrower at the bottom than at the top in what is known as the xe2x80x9cKeystonexe2x80x9d effect.
Radial distortion occurs when an image pixel is displaced from its ideal position along a radial axis of the image. Because an image has the largest field angles in the display comers, the comers exhibit worse radial distortion than other display areas. Radial distortion includes barrel distortion, where image magnification decreases towards the corners, and pin cushion distortion, where the magnification increases towards the corners. Lens related distortions including radial distortion can cause image deformation. Distortion can also result from non-flat screens or the Earth""s magnetic field.
Image modulators (such as 110, 112, 114) have a fixed number of pixels spaced uniformly in a pattern. This type of uniform pattern is called an affinity-mapped display. Projecting an image from an image modulator to a display screen deforms the uniformity of pixel spacing. In other words, pixels are not correlated one-to-one from the image modulator to the display screen. Therefore, some screen display regions have more image modulator pixels than screen pixels while other screen display regions have fewer image modulator pixels than screen pixels.
For panoramic displays, motion artifacts appear where image objects move near the edges of curved screens. Even when a flat screen projection is motion-adaptive filtered, the difference in the distances of objects from the projector causes an apparent motion of moving objects on a curved screen. Additionally, extremely large curved screens can achieve necessary resolution and brightness only with film projectors.
Multiple camera systems are commonly used to improve display quality on curved screen displays. For example, two cameras record overlapping halves of a scene to improve output. A layered coding technique may include a standard MPEG-2 stream as a base layer and enhancement information as a supplemental layer. Even if the two views are from slightly different angles, the compression ratio for the two camera views combined is less than the total compression ratio would be if each view were captured and compressed independently. Additionally, the second camera can provide a view that may be occluded from the first camera. Systems using additional camera angles for different views can provide additional coded and compressed data for later use. Multiple camera systems can also compensate for the limited focal depth of a single camera and can substitute for the use of a depth-finding sensor which senses and records depth information for scenes. Image processing can improve the outputs of multiple camera systems.
Stereoscopic photography also uses multi-camera systems in which a first camera records a left-eye view and a second camera records a right-eye view. Because camera lenses focus at a certain distance, one camera uses one focal plane for all objects in a scene. A multi-camera system can use multiple cameras each to capture a different focal plane of a single scene. This effectively increases the focal depth. Digital image processing can further improve focusing for these multi-camera systems.
Types of three dimensional binocular display systems include anaglyph displays, frame sequence displays, autostereoscopic displays, single and multi-turn helix displays. These normally have multiple camera data channels. Anaglyph systems usually require a user to wear red and green glasses so that each eye perceives a different view. Frame sequencing systems use shutter glasses to separate left and right views. Autostereoscopic displays use lenticular lenses and holographic optical elements. Single or multi-turn helix displays use multiple semi-transparent display screens which can be seen by multiple observers without special glasses. Multiple camera data channel systems can benefit from image processing.
Each R, G, and B color component has different intensity values which are digitally represented by a number of bits. For example, if 8 bits represent each R, G, and B color component, then each component has 256 (=28) intensity values from 0 to 255. Changing the intensity value of a color component in an ideal digital device from a number X, for example, to a number Y, takes just as long regardless of the Y value. Consequently, changing a color component value from 2 to3 takes as long as changing the value from 2 to 200. However, because of the nature of LCD image modulator pixels, the transitions for modulating light intensities are not purely digital, and various analog distortions remain.
Therefore, for all the foregoing reasons, what is needed is an image processing system to effectively enhance display quality and thereby provide the best possible visual images.
The present invention relates generally to image processing for a digital display system, and relates more particularly to an apparatus and method for performing image transforms and multi-frame processing of input images to produce high-quality output images. The image processing and digital display system are useful for DTV displays and electronic theatres, and can process different types of data inputs including analog, digital, compressed bitstream and coded bitstream display images.
In one embodiment of the present invention, the image processing uses the input data, along with the known characteristics of the particular display system, and advantageously performs geometric transformation to produce pre-compensated output images that are stored to a display modulator. The pre-compensated display modulator images are then projected to a display screen where the foregoing geometric transformation allows the displayed images to accurately portray the input images.
The geometric transformation produces high quality projection images through redefining the spatial relationship between image pixels to correct for image defocus, image distortion and misalignment and rotation of image modulators. In order to perform pre-compensation, improved resolution image representations are used, including high definition input images and input images enhanced through multiframe reconstruction. Multiframe reconstruction uses multiple input images, along with motion tracking information that ties the images together, to produce a higher resolution representation of each input image. The motion tracking information can either be provided as part of an input bitstream or produced by the system in a motion estimation module.
The geometric transformation may also provide special functions such combining multiple input images into a single output image, texture mapping an output image or producing specially constructed outputs for panoramic and 3D displays. The special information for display objects may either be extracted by the image processing system or for better results, the object information is provided as specially coded information in a bitstream. The present invention thus effectively and efficiently performs image transforms and multi-frame processing of input images to produce high-quality output images.